Add support for Ephemeral TaskLists #1449

natemort · 2025-08-07T18:16:01Z

Update Sessions to use Ephemeral TaskLists behind a feature flag. This ensures that the per-host TaskList is automatically removed once it is no longer used.

This should only be enabled once the server fully supports Ephemeral TaskLists as it will otherwise return errors for the unknown TaskListKind.

What changed?

Added support for dispatching session activities to ephemeral TaskLists and polling Ephemeral TaskLists

Why?

To support automatically removing session and integration test TaskLists when they're no longer used.

How did you test it?

Unit/integration test. Will run through canary tests before enabling.

Potential risks

Errors in task polling would significantly impact users, so this is a high risk change.

Detailed Description
See above.

Impact Analysis

Backward Compatibility:
Forward Compatibility: [Analysis of forward compatibility]

Testing Plan

Unit Tests: Unit tests
Persistence Tests: Covered in the server. See Support dispatching activities with a specific TaskListKind cadence#7157
Integration Tests: Added here, additionally present in the server.
Compatibility Tests: [Have we done tests to test the backward and forward compatibility?]

Rollout Plan

What is the rollout plan? Enable where fully rolled out on the server
Does the order of deployment matter? Server must support this feature before it is enabled clientside, otherwise poll requests will return an error.
Is it safe to rollback? Does the order of rollback matter? Safe to roll back
Is there a kill switch to mitigate the impact immediately? Disabling the flag.

codecov · 2025-08-08T15:12:11Z

Codecov Report

❌ Patch coverage is 96.07843% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.11%. Comparing base (2737bd0) to head (cd4590f).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/internal_event_handlers.go	50.00%	2 Missing ⚠️

❌ Your project check has failed because the head coverage (83.11%) is below the target coverage (85.00%). You can increase the head coverage or adjust the target coverage.

Files with missing lines	Coverage Δ
internal/activity_task_handler.go	`96.77% <100.00%> (ø)`
internal/client.go	`76.06% <100.00%> (+0.20%)`	⬆️
internal/internal_activity.go	`62.15% <ø> (ø)`
internal/internal_task_handlers.go	`81.92% <100.00%> (+0.03%)`	⬆️
internal/internal_task_pollers.go	`86.07% <100.00%> (ø)`
internal/internal_utils.go	`77.70% <ø> (ø)`
internal/internal_worker.go	`82.59% <100.00%> (+5.98%)`	⬆️
internal/internal_worker_base.go	`78.41% <ø> (+6.16%)`	⬆️
internal/workflow.go	`76.58% <100.00%> (+0.14%)`	⬆️
internal/workflow_replayer.go	`77.82% <100.00%> (+0.30%)`	⬆️
... and 2 more

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2737bd0...cd4590f. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

shijiesheng · 2025-09-16T17:32:05Z

internal/client.go

 	if options != nil {
 		return FeatureFlags{
 			WorkflowExecutionAlreadyCompletedErrorEnabled: options.FeatureFlags.WorkflowExecutionAlreadyCompletedErrorEnabled,
 			PollerAutoScalerEnabled:                       options.FeatureFlags.PollerAutoScalerEnabled,


just a comment: this flag is no longer used and I'll delete it in the next PR

shijiesheng · 2025-09-16T18:13:55Z

test/integration_test.go

+
+func (ts *IntegrationTestSuite) TestSession_Ephemeral() {
+	// Ephemeral TaskList is enabled by the test name
+	ts.TestSession()


I wonder if we can describe tasklist to ensure the tasklist is gone.

The TTL is rather high unfortunately. We can't really validate that behavior without inspecting the database directly.

test/integration_test.go

shijiesheng · 2025-09-16T18:31:38Z

test/workflow_test.go

+	}
+	defer workflow.CompleteSession(sessionCtx)
+	var expected string
+	if err = workflow.ExecuteActivity(ctx, "Prefix_ToUpper", "hello").Get(ctx, &expected); err != nil {


should be sessionCtx

Good catch, thanks

shijiesheng · 2025-09-16T21:37:33Z

test/integration_test.go

+func (ts *IntegrationTestSuite) TestSession_Ephemeral() {
+	// Ephemeral TaskList is enabled by the test name
+	ts.TestSession()
+}
+


this could do verification

Suggested change

func (ts *IntegrationTestSuite) TestSession_Ephemeral() {

// Ephemeral TaskList is enabled by the test name

ts.TestSession()

}

func (ts *IntegrationTestSuite) TestSession_Ephemeral() {

// Ephemeral TaskList is enabled by the test name

execution, err := ts.executeWorkflow("test-session", ts.workflows.Session, nil)

ts.NoError(err)

taskList := ts.getActivityTaskList(execution)

ts.Equal(shared.TaskListKindEphemeral, taskList.GetKind())

taskList = ts.describeTaskList(taskList.GetName())

ts.Equal(shared.TaskListKindEphemeral, taskList.GetKind())

}

func (ts *IntegrationTestSuite) getActivityTaskList(execution *workflow.Execution) *shared.TaskList {

iter := ts.libClient.GetWorkflowHistory(context.Background(), execution.ID, execution.RunID, false, shared.HistoryEventFilterTypeAllEvent)

for iter.HasNext() {

event, err := iter.Next()

ts.Require().NoError(err)

if event.GetActivityTaskScheduledEventAttributes() != nil {

return event.GetActivityTaskScheduledEventAttributes().GetTaskList()

}

}

return nil

}

func (ts *IntegrationTestSuite) describeTaskList(taskListName string) *shared.TaskList {

descResp, err := ts.libClient.DescribeTaskList(context.Background(), taskListName, shared.TaskListTypeActivity)

ts.Require().NoError(err)

return descResp.GetTaskList()

}

Discussed offline and added. The workflow now returns the TaskList the activity ran on, which we can check.

Update Sessions to use Ephemeral TaskLists behind a feature flag. This ensures that the per-host TaskList is automatically removed once it is no longer used. This should only be enabled once the server fully supports Ephemeral TaskLists as it will otherwise return errors for the unknown TaskListKind. Signed-off-by: Nathanael Mortensen <[email protected]>

natemort requested review from 3vilhamster, Groxx, demirkayaender, dkrotx, jakobht, shijiesheng and taylanisikdemir as code owners August 7, 2025 18:16

natemort force-pushed the ephemeral branch from 4e1ae76 to 33955d2 Compare August 7, 2025 20:46

natemort force-pushed the ephemeral branch 3 times, most recently from 4ef39f0 to 79c12c6 Compare September 15, 2025 22:26

shijiesheng reviewed Sep 16, 2025

View reviewed changes

test/integration_test.go Outdated Show resolved Hide resolved

shijiesheng reviewed Sep 16, 2025

View reviewed changes

davidporter-id-au approved these changes Sep 16, 2025

View reviewed changes

shijiesheng reviewed Sep 16, 2025

View reviewed changes

natemort force-pushed the ephemeral branch 6 times, most recently from c4aa7f0 to cd46896 Compare September 17, 2025 15:11

shijiesheng approved these changes Sep 17, 2025

View reviewed changes

natemort force-pushed the ephemeral branch from cd46896 to 97fefdd Compare September 17, 2025 16:17

natemort merged commit 7e2e45d into cadence-workflow:master Sep 17, 2025
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for Ephemeral TaskLists #1449

Add support for Ephemeral TaskLists #1449

natemort commented Aug 7, 2025 •

edited

Loading

Uh oh!

codecov bot commented Aug 8, 2025 •

edited

Loading

Uh oh!

shijiesheng Sep 16, 2025

Uh oh!

shijiesheng Sep 16, 2025

Uh oh!

natemort Sep 17, 2025

Uh oh!

Uh oh!

shijiesheng Sep 16, 2025

Uh oh!

natemort Sep 17, 2025

Uh oh!

shijiesheng Sep 16, 2025

Uh oh!

natemort Sep 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-func (ts *IntegrationTestSuite) TestSession_Ephemeral() {
-	// Ephemeral TaskList is enabled by the test name
-	ts.TestSession()
-}
+func (ts *IntegrationTestSuite) TestSession_Ephemeral() {
+	// Ephemeral TaskList is enabled by the test name
+	execution, err := ts.executeWorkflow("test-session", ts.workflows.Session, nil)
+	ts.NoError(err)
+	taskList := ts.getActivityTaskList(execution)
+	ts.Equal(shared.TaskListKindEphemeral, taskList.GetKind())
+	taskList = ts.describeTaskList(taskList.GetName())
+	ts.Equal(shared.TaskListKindEphemeral, taskList.GetKind())
+}
+func (ts *IntegrationTestSuite) getActivityTaskList(execution *workflow.Execution) *shared.TaskList {
+	iter := ts.libClient.GetWorkflowHistory(context.Background(), execution.ID, execution.RunID, false, shared.HistoryEventFilterTypeAllEvent)
+	for iter.HasNext() {
+		event, err := iter.Next()
+		ts.Require().NoError(err)
+		if event.GetActivityTaskScheduledEventAttributes() != nil {
+			return event.GetActivityTaskScheduledEventAttributes().GetTaskList()
+		}
+	}
+	return nil
+}
+func (ts *IntegrationTestSuite) describeTaskList(taskListName string) *shared.TaskList {
+	descResp, err := ts.libClient.DescribeTaskList(context.Background(), taskListName, shared.TaskListTypeActivity)
+	ts.Require().NoError(err)
+	return descResp.GetTaskList()
+}

Add support for Ephemeral TaskLists #1449

Add support for Ephemeral TaskLists #1449

Conversation

natemort commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

shijiesheng Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

shijiesheng Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

natemort Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shijiesheng Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

natemort Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

shijiesheng Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

natemort Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

natemort commented Aug 7, 2025 •

edited

Loading

codecov bot commented Aug 8, 2025 •

edited

Loading